Test Automation Framework

The book has now been published and the content of this chapter has likely changed substanstially.
Please see page 298 of xUnit Test Patterns for the latest information.
How do we make it easy to write and run tests written by different people?

We use a framework that provides all the mechanisms needed to run the test logic so the test writer only needs to provide the test-specific logic.

Sketch Test Automation Framework embedded from Test Automation Framework.gif

Writing and running automated tests involves a number of different steps. Many of these steps are the same for every test. If every test had to include an implementation of these steps writing automated tests would be very tedious, time consuming, error prone and expensive.

Using a Test Automation Framework is a way to minimize the effort of writing Fully Automated Tests (see Goals of Test Automation on page X).

How It Works

We build a framework that implements all the mechanisms required to run suites of tests and record the results. These mechanisms include the ability to find individual tests, assemble them into a test suite to run, execute each test in turn, verify expected outcomes have occurred, collect and report any test failures or test errors and to clean up when they do occur. The framework provides a way to plug in and run the test-specific behavior that test automaters write.

Why We Do This

Building Fully Automated Tests that are repeatable and robust is a lot more involved than just writing a test script that invokes the system under test (SUT). We need to be able to handle both success cases and errors, both expected and unexpected. We need to be able to set up and tear down test fixtures. We need to be able to specify what test(s) to run. We also need to be able to report on the results of having running a suite of tests.

The amount of effort involved can act as a serious deterrent to automation of tests. We can reduce the entry cost significantly by providing a framework that implements the most common functionality. Then, the only entry cost is learning to use the framework. This in turn can be reduced if the framework implements a common protocol such as xUnit which makes it easier for us to learn the second or third such framework once we have experience with the first.

Using a framework also helps isolate the implementation of logic required to run the tests from the logic of the tests. This can help reduce Test Code Duplication (page X) and Obscure Test (page X)

Implementation Notes

There are many kinds of Test Automation Frameworks available, both commercial and open-source. They can be divide into two main categories; one group is the "Robot User" class of test tools while the other is the Scripted Test (page X) category which can be further subdivided into the xUnit family of Test Automation Frameworks and the Data-Driven Tests (page X) family.

Variation: Robot User Test Frameworks

There are a large number of third-party test automation tools that are designed to test application via the user interface. Most of them use the "Record and Playback" test metaphor; this is very seductive marketing because it makes test automation seem as simple as running some tests manually while recording the test session. THe consist of two major parts, the "Test Recorder" which is used to monitor and record the interactions between the user and the SUT and the "Test Runner" which is used to execute the Recorded Tests (page X). Most of these test automation tools are also frameworks into which a number of "Widget Recognizers" are plugged. Most commercial tools come with a gaggle of widget recognizers built in.

Variation: The xUnit Family of Test Automation Frameworks

Most unit testing tools belong to the xUnit family of testing frameworks designed for automating Hand-Scripted Tests (see Scripted Test) . xUnit has been ported to (or developed from scratch for) most current programming languages.The xUnit family of unit testing frameworks consists of several major components. The most visible is the Test Runner (page X) which can be invoked from the command line or as a Graphical Test Runner (see Test Runner). It builds the Testcase Objects (page X), collects them into Test Suite Objects (page X) and invokes each of the Test Methods (page X). The other major component is the library of built-in Assertion Methods (page X) that are used within the Test Methods to specify the expected outcome of each test.

Variation: Data-Driven Test Frameworks

A Data-Driven Test framework provides a way to plug in interpreters that know how to execute a specific kind of test step. This, in effect, extends the format of the input file with new "verbs" and objects. They also provide a "Test Runner" that reads in the file, passes control to the plug-ins when their corresponding data formats are encountered and to keep track of statistics for the test run. The most notable member of the Data-Driven Test Frameworks family is Fit. It provides the means to write tests in tabular form and to "plug in" fixture classes that know how to interpret specific formats of tables.

Example: Test Automation Framework

Test Automation Framework that looks somewhat different in each of the possible ways to automate tests. Please refer to Recorded Test, Scripted Test and Data-Driven Test for examples of the respective Test Automation Frameworks.